Mutation Coverage Estimation for Model Checking

When engineers design a system, there is always a question about how exhaustive the system has been examined to be correct. Coverage estimation provides an answer to this question in testing. A model checker verifies a design exhaustively, and proves the satisfaction of property specifications. However, people have noticed that design errors exist even after model checking is done, which goes to show that the question “How complete is the model checking once done?” is still left relatively unaddressed by model checkers, except for some state-based coverage metrics and the coverage estimator for symbolic simulation in RED. As a more complete solution, we propose several structural mutation models and coverage metrics to cover different design aspects in a state graph and to estimate the completeness of model checking, respectively. Once a system state graph satisfies a given set of property specifications, we estimate the coverage of completeness for the set of properties by applying some mutations to the state graph and checking if the given set of properties is sensitive to the mutation. Our experiences on five application examples demonstrate how the proposed coverage estimation methodology helps verification engineers to find the uncovered hole.